A Library for Implementing the 
Multiple Hypothesis Tracking Algorithm 



David Miguel Antunes^, David Martins de Matos^, and Jose Caspar^ 

^ Institute for Systems and Robotics, LF'F, davidmiguel [at] antimes.net 

^ L^F - INESC-ID, david.matos [at] 12f.inesc-id.pt 
^ Institute for Systems and Robotics, IST/UTL, jag [at] isr.ist.utl.pt 



Abstract. The Multiple Hypothesis Tracking (MHT) algorithm is known 

to produce good results in difficult multi-target tracking situations. How- 
ever, its implementation is not trivial, and is associated with a signifi- 
cant programming effort, code size and long implementation time. We 
propose a library which addresses these problems by providing a domain 
independent implementation of the most complex MHT operations. We 
also address the problem of applying clustering in domain independent 
manner. 



1 Introduction 

The Miiltiplc Hypothesis Tracking (MHT) algorithm, proposed by Reid [1] is 
fundamental in the multi-target tracking field with many applications in differ- 
ent areas. And, even though its significant computational complexity inhibited 
its utilization for some time, continuous increases in the processing power of 
computers accompanied by a reduction in their cost, along with some algorith- 
mic improvements [2] , make the implementation of the MHT feasible nowadays 
[3]. The MHT is now the preferred method for difBcult tracking problems [4]. 

Although the MHT presents advantages when compared to other methods 
its implementation is not trivial, and requires significant programming effort, 
extensive code size, and debugging effort [5]. To address the difficulty in im- 
plementing the MHT, we propose and describe a library which can be used to 
ease this task. Because the MHT is applied in multiple domains, with variations 
on the algorithm, such as group tracking [6], the library should not enforce a 
particular type of MHT implementation. Therefore, it only addresses the man- 
agement of the multiple hypotheses, and not the probability modeling, movement 
models, sensor models, and other aspects which may vary across different MHT 
implementations . 

Cybenko et al. [7] identify less usual tracking domains, such as computer se- 
curity, or social networks. Because the proposed library is not tied to a particular 
domain, it may be used to apply tracking to domains like the ones identified by 
these authors. 



As noted in the seminal work of Reid [1] , an efficient MHT implementation 
must perform clustering. Clustering consists in dividing the tracking problem 
into independent sub-problems which can be solved separately. Reid identified 
the necessary steps to implement clustering in the MHT. However, because the 
proposed library is independent of the specific domain of the application, and 
the clustering method described by Reid is specific for a particular domain, a dif- 
ferent, more general clustering method is required. We propose such a clustering 
method for the library. 

The work on Process Query Systems (PQS) [8] by various persons of the 
Dartmouth College is close in intent with this one. PQS allow the modeling of 
processes with states, dynamics, and observablcs in a domain independent man- 
ner. The PQS framework is then able to detect and track the evolution of several 
processes in the environment. However, the PQS framework does not seem to 
provide clustering, and is tied to the notion of process which may not be appro- 
priate for some multiple hypothesis applications, such as Multiple Hypothesis 
Situation Analysis [9]. Jean Roy et al. proposed a domain independent cluster- 
ing method for the MHT [10], however he does not provide any description of a 
domain independent MHT library. 

The main contribution of this paper is the description of a library which can 
be used to simplify and speed up the implementation of the MHT independently 
of the application's domain, and a method to provide the ability of clustering in 
such a library. 

The proposed library was applied to several problems which arc discussed, 
one of which is detailed in section 6. Section 2 gives an overview of the proposed 
library, section 3 details the clustering method, section 4 describes the modeling 
of information, and the library's operations and data-structures are described in 
section 5. 

More information and resources on the proposed library are available at 
http : //www .multiplehypothesis . com. 

2 MHT Overview 

The usual flow diagram for the MHT is presented in figure 1. First, the applica- 
tion receives a new set of measurements from the sensor. Each measurement is 
either associated with an existing cluster or else a new cluster is created. When 
a measurement is associated with more than one cluster they are joined and the 
resulting cluster is pruned. Afterwards, new hypotheses arc generated for each 
set of measurements associated with the same cluster, followed by cluster tree 
pruning. Finally, the clusters may be decomposed according to the rule: if two 
tracks share a common measurement, they must be on the same cluster. 

The operations of the library (shown in figure 2) are similar to the ones of the 
traditional MHT. There is, however, a clear separation of responsibilities between 
the application and the library. The main responsibility of the application is 
the hypothesis generation, as it is domain dependent. The library handles the 
complicated operations which must modify complex data-structures, freeing the 
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Fig. 1. Traditional MHT. Fig. 2. Proposed library workflow, and an abstract example 
of how the operations affect the maintained data structures 
(right). 



library user from the most difficult and time consuming programming tasks. 
How the hypothesis generation is performed in detail, as well as other necessary 
operations, will be further discussed in the rest of the paper. 



3 Clustering Method 

Before further detailing the library, the proposed clustering method will be in- 
troduced. 

Because the library is independent of the application domain it only sees 
generic information. To ease application development, this information is sepa- 
rated in history and state of the world. It is not necessary for the application to 
take advantage of this separation, however it has been found useful in structur- 
ing the development of tracking applications. For now, consider that the library 
maintains a set of information 7, constituted of pieces of information ii , ^2, • 
The specific modeling of information and its separation in history and state of 
the world, will be discussed later, in section 4. 

Clustering on the library is based on two rules. The first one is the following: 
If ii and 12 where generated at the same hypothesis generation, then they must 
remain in the same cluster. And the second rule is: //, to assert i\, one needs to 
know 12, then both must remain in the same cluster. The algorithm operations 



which concern chistering try to form the maximum number of clusters, while 
still respecting these two rules. 

The validity of the proposed clustering rules will now be discussed. 




Fig. 3. Tracking scenario (left), and an abstract rep- Fig. 4. Applying the second clus- 
resentation of a possible hypothesis tree (right). tering rule. 



3.1 Validity of the First Rule 



In the MHT, in an hypothesis generation, no more than one hypothesis can 
be correct, and exactly one of the hypotheses will be considered correct, i.e. 
only one will survive pruning in the long run. Consider the scenario in figure 
3, where target T is being tracked. Measurements are denoted by numbers 1 
to 7. At the right there is an abstract representation of a possible hypothesis 
tree (or cluster). The first rule dictates that the information inside A, B, and 
C, cannot be separated to different clusters. A separation of this information 
could lead to inconsistent situations. For example, consider FA (6) (meaning 
measurement 6 is a false alarm) and T(6) (meaning measurement 6 is a detection 
of target T) on B are separated to different clusters. Then, there would be a 
global'' hypothesis where both would be true that is, a measurement could be 
both a false alarm and a detection of target T at the same time. And there 
would also be a global hypothesis where none would be present, resulting in 
any information related to measurement 6 disappearing, if only this hypothesis 
survived pruning. Thus, the rule first rule ensures that no more and no less than 
one of the pieces of information generated in one hypothesis generation will be 
present in each possible global hypothesis. 

From now on, the sets of information which must be on the same cluster, 
such as A, B, and C will be denoted as constraints. 



* Every cluster contains information only on a specific part of the world. A global 
hypothesis contains information on all of the world, and it is built by selecting an 
hypothesis from every cluster [4]. 



3.2 Validity of the Second Rule 



To apply the second rule, the library needs to know which information the appli- 
cation needs to generate a certain hypothesis. In practice, this is implemented in 
the following manner. Before generating new hypotheses, the application iden- 
tifies the information which will be required to generate those hypotheses (a 
subset of all the information the library is storing) and requests it from the li- 
brary. When the application generates new hypotheses for a leaf in a cluster's 
hypothesis tree, the library provides it with the subset of the required informa- 
tion which is true in that leaf. According to the second rule, the information 
provided by the library and the information generated by the application must 
be kept on the same cluster. 

The constraints introduced by the first rule alone are not sufficient to cor- 
rectly split clusters. Consider the example in figure 3. Although the information 
in constraints A, B, and C cannot be separated, the hole constraints may be 
separated to different clusters. In which case, for example, the following global 
hypothesis would be possible: { T(5) , FA(6) , T(7) }. This hypothesis is incorrect 
as T(7) only makes sense with T(6) . With only the first rule it is possible that in 
a global hypothesis there are two pieces of information incompatible with each 
other, or that there is a missing piece of information (such as T(6)). However, 
when generating hypotheses, the application must (and needs to) request any 
piece of information which would prevent the generated hypotheses from mak- 
ing sense, or any piece of information which it requires to know to generate the 
hypotheses (such as T(6)). Then, the constraints introduced by the second rule 
ensure that there are no incorrect global hypotheses. In the example of figure 
3, the application would request the information on targets near measurement 
7 (corresponding to T(6)) and, applying the second rule, constraint E would be 
introduced. The case for constraint D is similar. 



4 Information Modeling 

As the library is generic and it is intended to be applicable in multiple domains 
the modeling of the information contained in the library is also generic. The 
library can store and manage two different types of information: events and 
facts. The library does not contain any predefined events nor facts, they are 
defined by the application in order to fit its needs. 

4.1 Events 

An event corresponds to a piece of history of the world, and contains information 
about what (possibly) happened in the world at some point in time. For exam- 
ple "ship 9372 moved to (1232.2, 293.6) at 15:53AM", or "sensor with id=173 
produced a false alarm at 04:26PM". 



4.2 Facts 



In order to request the necessary information from the Hbrary, prior to an hy- 
pothesis generation, the appHcation has to identify and find the events which will 
be relevant for that hypothesis generation. For example, following a detection at 
{x, y) by a ship's radar, it is necessary to identity the events which place a ship 
near (a;, y), as it might be the cause of that detection. And it may happen that 
a ship was detected in that area two minutes, half an hour, or even three hours 
ago. Thus, the application may need to search trough many events to find the 
ones it requires. The same happens if the application has to answer the question: 
"where is ship 9372?" or "how many ships are in the (xq, j/o, a^i, area?". 

To simplify the answer to these questions, the application may define facts. 
Whereas events arc part of the history of the world, facts represent the current 
state of the world. For example, when the application generates the event "ship 
9372 moved to (1232.2, 293.6) at 15:53AM", it may also generate the fact "ship 
9372 is in (1232.2, 293.6)". Contrary to events, facts do not usually include a 
timestamp, as a fact is true as long as it exists. And, while events are never 
deleted (unless they are pruned), facts have a limited lifetime. In the example, 
when the ship moves a new fact should be created with the new position, and 
the old one deleted. In the example, if there is a fact per ship, answering the 
aforementioned questions becomes simpler as there is one, and only one, fact for 
each ship which contains the most up-to-date information on the ship position, 
velocity, etc. 

5 The Librciry's Operations 

The library's main operations which will now be detailed are depicted in figure 
2, and they are: cluster joining, hypothesis generation, cluster splitting, and 
pruning. However, before detailing these operations, the data structures upon 
which they operate will be described. 

5.1 Data Structures 




Fig. 5. Example of the tree structure (or cluster). 



The fundamental data structure in the Ubrary is the hypothesis tree (or 
cluster). An example tree is depicted in figure 5. The events in a generated 
hypothesis can be treated in the same manner for most of the operations, so, they 
are aggregated in event groups. The nodes in the hypothesis tree are constituted 
of event groups (denoted by EG in figure 5), containing events (denoted by E). 
The tree also contains constraints (denoted by C) and, associated with each event 
group at the bottom of the tree, there is a structure denoted by leaf (denoted by 
L). Facts (denoted by F) are stored in the leaves. The library maintains several 
of these trees at the same time. 

Additionally there is a relation between each fact and an event group, not 
shown in the figure. For each generated hypothesis, its events are stored in a 
new event groiip and the facts added to a new leaf. Then, each of the facts is 
associated with the new event group. We say the fact "depends upon" the event 
group. This association dictates, for example, the fact's probability, which is the 
same that the probability of its associated event group. 

In section 3, a simplification was made by considering that the library only 
keeps "information". Now the clustering rules will be stated directly in terms 
of events, facts, event groups, and constraints. The first rule can be formulated 
as: In an hypothesis generation, a new constraint must be created containing 
all the new event groups. And the second rule corresponds to: When generating 
hypotheses for a leaf, a new constraint must be created with: the new event groups; 
the leaf's event groups containing requested events; and the leaf's event groups 
to which requested facts are associated. The event groups of a leaf are all the 
event groups which are in the path from the leaf up to the root of the hypothesis 
tree. 

5.2 Operations 

An overview of the operations depicted in figure 2, and which will be detailed 
next is as follows. After receiving a new measurement, the application needs to 
generate hypotheses on the origin and implications of the new measurement. 
The application knows all the facts and events contained in the library, and 
selects the ones which are relevant for the hypothesis generation. The library 
receives the relevant events and facts for that hypothesis generation and joins 
all the clusters which contain them into one single cluster, then, requests the 
application to generate new hypotheses for each leaf of the cluster, providing 
the application with the subset of the requested events and facts which are true 
according to that leaf. Afterwards the cluster is pruned and, if it can now be 
divided into clusters containing subsets of the events, facts, and constraints in 
the original one, while still complying to the clustering rules, then splitting is 
applied. 

Procedure 5.1.1, JoinClusters, details the method for joining different clus- 
ters. Pruning is applied each time two clusters are joined (the representation in 
figure 2 simplifies this by showing pruning only once, after cluster joining). The 
procedure UnifyConstraints is called by the previous one to guarantee that, 
for every constraint c in cluster c2 (see procedure 5.1.1) containing the set of 



events e, a constraint C exists in the final cluster which contains the set E of all 
the events of the clones of constraint c. Thus, \E\ = |e| x \cl.leaves\. If no events 
or facts are requested, a new cluster is created (cluster joining is not necessary). 

The procedure 5.1.3, HypGen, asks the application to generate hypotheses 
for each leaf of the cluster. When generating hypotheses for a leaf, it provides the 
application with the subset of the requested facts and events which are available 
in the leaf. The generation will result in the tree being augmented with new event 
groups and new leaves. The HypGen procedure also creates the appropriate new 
constraints and prunes the resulting tree. 

Several pruning strategies may be applied. The pruning operation can limit 
the number of leaves, the depth of the tree, or follow other application specific 
strategy. When a leaf is pruned, procedure 5.1.4, RemoveLeaf, is called to 
remove it from the cluster. When a brach of the tree is pruned, RemoveLeaf 
is called for every leaf coming from that branch. 

Cluster splitting can be performed if non-intersecting sets of constraints are 
found. If constraints cl and c2 contain a common event group they intersect. For 
each set of independent constraints, C, a new cluster, nCluster, can be created 
by cloning the original cluster. In nCluster, the events in event groups which 
are not contained in any c G C are deleted. Also, only facts associated with 
event groups in c G C are kept, and nCluster only contains the constraints in 
C. Procedure 5.1.5, Split, details this operation. 

Procedure 1 Join clusters with required events or facts 
procedure JoiNCLUSTERS(re(7£^wenfs, reqFacts) 
Find the clusters, clusters, containing required events or facts; 
Select a cluster cl e dusters; 
For each c2 ^ cl in clusters: 
clones ^ {}; new Leaves ■<—{}; 
For each leaf in cl.leaves: 
clone ■h- clone of c2; Add clone to clones; 
Add c2.rootEventGroup to leaf.eventGroup children; 
For each cLeaf in clone.leaves: 
Add cLeaf to newLeaves; Add leaf .facts to cLeaf .facts; 
cLeaf.proh cLeaf.prob x leaf.prob; 
Delete c2; cl.leaves newLeaves; UnifyGonstraints (cl, clones); 
Prune cl; 
Return cl; 

Procedure 2 Connect constraints in cloned clusters 
procedure UnifyGonstraints (cl, clones) 
For i 1 up to \clones[l].constraints\: 

Create constraint newC; 

For each clone in clones: 
Add all event groups in clone.constraints[i] to newC .eventCroups; 

Add newC to cl. constraints; 



Procedure 3 Generate hypotheses 



procedure HYPGEN(dwsfer, reqEvents, reqFacts) 
newEventGroups ^ {} 
For each leaf in cluster: 

provFacts leaf .facts fl reqFacts; provEvents leaf .events fl 
reqEvents; 

Create a new constraint, cstrnt, with the event groups containing provided 

events, and the event groups to which provided facts arc associated; 

Request the apphcation to generate hypotheses, providing it with 
provEvents and provFacts; 
For each generated hypothesis, hyp: 
Create a new event group, nEventGroup, with hyp.events; 
Create a new leaf, nLeaf, with hyp. facts U {leaf .facts \ provFacts); 
Add nLeaf to cluster.leaves; 

Add nEventGroup to leaf.eventGroup children, to newEventGroups, 
and to cstrnt; 

Set nLeaf .probability to hyp. probability * leaf .probability; 
Remove leaf from cluster; 
Create a new constraint with newEventGroups; Normalize leaves probabil- 
ity; Prune cluster; 

Procedure 4 Remove leaf from cluster 

procedure REMOVELEAF(dM.ster, leaf) 

Remove leaf from cluster.leaves; parent leaf.eventGroup; 

While \parent. children] = do: 

Remove parent from all cluster. constraints, and from 
parent. parent. children; 

parent ^ parent. parent: 
Remove empty constraints in cluster. constraints; Normalize leaves probabil- 
ities; 

Procedure 5 Cluster splitting 
procedure SPLiT(duster) 

super C ousts {}; Let map be a map from constraints to sets of constraints; 
For each c in cluster. constraints: 

Let sC be a clone of c; Add sC to superC ousts; map[sC] { c }; 
For all sCi,sC2 € superConsts : {sCi.eventGroups fl sC2.eventGroups) ^ 
{}: 

Create new constraint, newConstr; Add newConstr to superConsts. 
newConstr.eventGroups -r- sCi.eventGroups U sC2.eventGroups; 
Remove sCi and sC2 from superConsts; 

map[newConstr] <— map[sCi] U map[sC2]; Remove sCi and SC2 from map; 
For each constraints in map keys: 

Create a new cluster by cloning cluster. Keep only constraints constraints, 
the events contained in event groups in constraints, and the facts associated 
with those event groups; 
Remove cluster; 



5.3 Further Notes 



In order to request events and facts from the hbrary for an hypothesis generation, 
the apphcation may obtain aU the events and facts the Hbrary contains and 
search for the ones of interest. However, a better approach is for the hbrary to 
notify the apphcation about changes to its available events and facts. Then the 
application can organize them in a way which speeds up searches. For example, 
the application may organize facts containing the position of ships so that it is 
efhcient to find all the ships in a certain area. 

The library does not keep the events which have a probability of 1. These are 
the events at the root of the clusters for which there is no uncertainty. Therefore, 
after a cluster is pruned, they should be sent to the application and removed 
from the cluster. Then, the application decides what to do with them (storing 
them in persistent storage, for example). 

Facts can be seen as a special events. They have only two differences: contrary 
to events, facts are deleted after they are provided to the application in an 
hypothesis generation (but they may be generated again); when the event group 
associated with a fact reaches the root of the tree, with probability of 1, it is not 
sent to the application, instead, a new cluster is created with only that fact. 

It may happen that, due to cluster splitting, a cluster is created containing 
only facts and/or events which will never be requested by application. For ex- 
ample a cluster which only contains false alarm events. Because it will never be 
requested it is wasting memory space. Thus the application should provide a 
predicate function which accepts the facts and events of a cluster and asserts if 
that cluster can ever be required for hypothesis generation. If not, then best K 
hypothesis pruning is applied to the cluster, with K= 1. 

6 Radcir Tracking Application 

The proposed library was implemented in Java and applied to several tracking 
problems. One of the library applications is a radar tracker simulator, developed 
to illustrate the capabilities of the library in a self-contained manner, so that 
other members of the research community may become rapidly familiarized with 
the library work-flow by analyzing the application code. This simulator will now 
be described in detail. 

The simulated radar is stationary and detects ships in a circular region 
around it. Every time the radar beam ends a full 360° turn the radar detec- 
tions are provided to the tracker. A Kalman Filter with a constant velocity 
model is used to predict the position of the targets, but the simulated targets 
have random velocity and direction changes. 

The problem was modeled with facts and events. Only one fact (named Tar- 
getPositionFact) was required to model the problem. This fact contains the 
target identifier number, the state of the target's Kalman Filter, and the time 
of the target's last detection (which is used to terminate the track if the target 
is not detected after a certain period of time). 



Fig. 6. Ground truth. 



Fig. 7. Radar. 



Fig. 8. Tracker output. 



To keep the tracking history four events were defined. A TracklnitiatedEvent 
containing the position and the identifier of the target which initiated the track. 
A TrackTerminatedEvent containing the identifier of the target whose track ter- 
minated. A TargetMovedEvent containing the initial and final positions of the 
target, and the target identifier. And a FalseAlarmEvent with the position of 
the detection which was considered a false alarm. All these events contain a 
timestamp. 

After the tracker receives a new set of measurements, when the radar com- 
pletes a full 360°turn, it needs to generate hypotheses. Only facts are required 
from the library for hypothesis generation. For this reason, the tracker can func- 
tion with or without events. Without events it does not keep history, but is 
faster. When generating hypotheses for a measurement if a target, represented 
by a TargetPositionFact, contains the measurement in its validation gate then 
the fact must be required (the measurement may be a detection of that target). 
Furthermore, if the same TargetPositionFact must be required for two mea- 
surements, their hypothesis generation must be made at the same time. Each 
measurement either is a false alarm, a new target detection, or a detection of an 
existing target. 

Due to clustering the tracker is able to track many targets at the same time, 
while still taking an acceptable time to process each set of detections coming 
from the radar. Figure 9 shows the time to process the detections from the radar 
with different number of targets, it grows almost in a linear fashion. 

The proposed library was implemented in Java and made available on-line 
together with the demonstration of the radar tracking application. 

7 Conclusions 

In this paper we address the problem of the complexity of implementing the MHT 
by proposing a library which implements much of the algorithm, leaving only the 
domain specific tasks to the application developer. And, even though the library 
is independent of the application domain, we provide a method for applying 
clustering, also domain independent, which is essential for an efficient imple- 
mentation of the MHT. Additionally, the library provides an explicit separation 
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Fig. 9. Time to process new detections. Error bars show standard deviation. 

of the state and history of the world, which can be very useful in structuring 

tracking applications. 

In order to demonstrate the effectiveness of the library a radar tracking ap- 
plication was developed using the library. Simulations have shown successful 
tracking of the targets with a computation time growing in an approximately 
linear manner with the number of targets, due to the clustering ability of the 
library. 
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